Merging pcp flows as they are assigned to a single virtual channel

ABSTRACT

A Network Flow Processor (NFP) integrated circuit receives, via each of a first plurality of physical MAC ports, one or more PCP (Priority Code Point) flows. The NFP also maintains, for each of a second plurality of virtual channels, a linked list of buffers. There is one port enqueue engine for each physical MAC port. For each PCP flow received via the physical MAC port associated with a port enqueue engine, the port enqueue engine causes frame data of the flow to be loaded into one particular linked list of buffers. Each port enqueue engine has a lookup table circuit that is configurable to cause multiple PCP flows to be merged so that the frame data for the multiple flows is all assigned to the same one virtual channel. Due to the PCP flow merging, the second number can be smaller than the first number multiplied by eight.

TECHNICAL FIELD

The described embodiments relate generally to MAC frame ingressprocessing circuits and methods.

SUMMARY

A Network Flow Processor (NFP) integrated circuit includes a pluralityof SerDes circuits and a Media Access Control (MAC) layer interfacecircuit. The SerDes and MAC layer interface circuit are configurableinto a first number of physical MAC ports. The NFP integrated circuitreceives, via each of the physical MAC ports, one or more PCP (PriorityCode Point) flows. A PCP flow is a flow of frames received onto the samephysical MAC port, where all the frames have the same PCP code value.Each PCP flow received via a particular physical MAC port has a relativepriority with respect to each other PCP flow received via that physicalMAC port. In addition to the SerDes circuits and the MAC layer interfacecircuit, the NFP integrated circuit further includes a plurality of portenqueue engines, a pipelined buffer memory, a plurality of port dequeueengines, and a single minipacket parallel bus. For each of a secondnumber of virtual channels, a corresponding linked list of buffers ismaintained in the memory. There is one port enqueue engine for eachphysical MAC port. For each PCP flow of frames received via the physicalMAC port associated with a port enqueue engine, the port enqueue enginecauses frame data of the frames of the PCP flow to be loaded intobuffers of one particular linked list of buffers. The port dequeueengine associated with the physical MAC port dequeues the linked list ofbuffers, thereby causing the frame data of the PCP flow or flows storedin the linked list of buffers to be output onto the single minipacketparallel bus. Accordingly, all incoming frame data from all the virtualchannels is output onto the same one minipacket parallel bus, with each256-byte minipacket having an associated virtual channel number thatindicates the virtual channel. Further network processing functionalityof the NFP integrated circuit receives the frame data from theminipacket bus and performs further network processing. This furthernetwork processing functionality implements the second number of virtualchannels through the remainder of the NFP integrated circuit.

In a first novel aspect, each port enqueue engine has a PCP Remap LUT(PRLUT) and associated circuitry that causes multiple PCP flows to bemerged so that the frame data for the multiple PCP flows is all assignedto the same one virtual channel. Accordingly, the frame data for themultiple PCP flows is loaded into the same one linked list of buffers inthe memory, where the linked list of buffers is the linked list ofbuffers for the virtual channel. Due to the PCP flow merging of flows ofa physical MAC port, the second number can be, and is in one embodiment,smaller than the first number multiplied by eight. The number of virtualchannels can be, and is in one embodiment, smaller than the number ofphysical MAC ports multiplied by eight (the number of PCP prioritylevels that can be defined by the three-bit PCP code value of incomingframes).

In a second novel aspect, the PCP Remap LUT (PRLUT) of a port enqueueengine does not cause PCP flows to be merged so that the frame data ofmultiple PCP flows is assigned to one virtual channel, but rather thePRLUT is configured so that the relative priorities of the PCP flows arereordered and changed (with respect to one another) as the PCP flows areassigned to virtual channels. For example, a higher priority PCP flowwhose PCP value is a larger value is assigned to a lower priorityvirtual channel having a lower virtual channel numbers, whereas a lowerpriority PCP flow whose PCP value is a smaller value is assigned to ahigher priority virtual channel having a higher virtual channel number.In this example, within the virtual channels for a given physical MACport, the higher the virtual channel number is the higher the priorityof the virtual channel is.

In one embodiment, a PRLUT is configured to carry out PCP flow mergingon some PCP flows, and to also perform PCP flow reordering. In anotherembodiment, a first port enqueue engine causes PCP flow merging tooccur, whereas a second port enqueue engine causes PCP reordering tooccur. The PRLUTs of the port enqueue engines are independentlyconfigurable.

In a third novel aspect, each linked list of buffers that stores framedata is not actually a single linked list of buffers, but rather is adual linked list of buffers. In addition, a free buffer linked list ofbuffers is also a dual linked list of buffers. The dual linked lists ofbuffers are maintained by a link manager. The term “linked list ofbuffers” as the term is used here refers to a linked list of queueelements that stores buffer identification values (buffer IDs) alongwith the corresponding buffers that are identified by the buffer IDs.The link manager maintains, for each such dual linked list of buffers, afirst head pointer queue element H1, a second head pointer queue elementH2, a first tail pointer queue element T1, a second tail pointer queueelement T2, a head pointer active bit, and a tail pointer active bit.The first head pointer queue element and the first tail pointer queueelement are used to maintain the first linked list of the dual linkedlist. The second head pointer queue element and the second tail pointerqueue element are used to maintain the second linked list of the duallinked list. When a sequence of values (a sequence of buffer IDs) ispushed into the dual linked list of buffers, odd values of the sequenceare pushed into the first tail pointer queue element so that odd valuesare stored in the first linked list of buffer elements, whereas evenvalues of the sequence are pushed into the second tail pointer queueelement so that even values are stored in the second linked list ofbuffer elements. The tail pointer active bit indicates which tailpointer queue element will receive the next value to be pushed. Thevalue of the tail pointer active bit is toggled from push to push. Whena sequence of values is popped out of the dual linked list of buffers,odd values of the sequence are popped off the first head pointer queueelement, whereas even values are popped off the second head pointerqueue element. The head pointer active bit indicates which head pointerqueue element will be popped next. The value of the head pointer activebit is toggled from pop to pop.

In one example, the memory that stores queue elements other than thehead and tail pointer queue elements is a pipelined memory. Thepipelined memory has a read access latency time for reading the valuestored in a queue element. Due to the pipelined nature of the memory,however, the dual linked list system can pop the dual linked list ofvalues and supply dequeued values at a sustained rate of more than onevalue per the read access latency time. For example, if the dual linkedlist system is popped twice in rapid succession, then the pipelinedmemory will be performing multiple read operations of multiple queueelements at a given time with the read operations following each otherin sequence through the stages of the pipelined memory. The first headpointer queue element is popped, and then a memory read is initiated toreplenish the first head pointer queue element. Before this replenishinghas been completed, the second pop of the dual linked list occurs,thereby resulting in a pop of the second head pointer queue element. Avalue can be popped out of the second head pointer queue element at atime when the memory is in the process of being read to replenish thefirst head pointer queue element. A second read of the memory is alsoinitiated to replenish the second head pointer queue element. At thistime, both the first and second read operations are occurringsimultaneously, albeit with the first read operation leading the secondthrough the various stages of the pipelined memory.

The use of multiple linked lists to realize a faster access singlelinked list is extendable. For example, three different linked lists canbe maintained together by the link manager in order to realize a triplelinked list. Likewise, four different linked lists can be maintainedtogether by the link manager in order to realize a quadruple linkedlist, and so forth. The values stored in these linked lists aretypically buffer IDs, where each such buffer ID identifies acorresponding buffer.

In a fourth novel aspect, the link manager stores a predetermined andpre-configured “overflow threshold value” for virtual channel. The linkmanager also maintains, for each virtual channel, a buffer count wherethe buffer count is the number of buffers (in the linked list of buffersfor the virtual channel) that currently store frame data. In addition,in this fourth novel aspect, PCP flow merging and/or reordering wasperformed on enqueue. As frames are received on the NFP integratedcircuit and are stored in a linked list of buffers, if the buffer countfor a virtual channel is detected to exceed the “overflow thresholdvalue” for a virtual channel whose originating PCP flows were merged,then a PFC (Priority Flow Control) pause frame is generated wheremultiple ones of the priority class enable bits are set to indicate thatmultiple PCP flows should be paused. The setting of a priority classenable bit in the PFC pause frame is an indication to a receiver of thePFC pause frame that the PCP flow corresponding to that priority levelshould be paused. For the particular virtual channel that is determinedto be overloaded, an Inverse PCP Remap LUT (IPRLUT) circuit in the portenqueue engine performs inverse PCP mapping, including inverse PCPmerging and/or inverse PCP reordering, and outputs a multi-bit value.The multi-bit value indicates each of those PCP flows that is associatedwith the overloaded virtual channel. For each PCP flow identified inthis way, the corresponding bit in an 8-bit priority class enable vectoris set. The 8-bit priority class enable vector is supplied by the portenqueue engine to its corresponding physical MAC port. The physical MACport in turn uses the 8-bit priority class enable vector to generate thePFC pause frame so that the appropriate multiple enable bits are set inthe pause frame. The physical MAC port then outputs the PFC pause framefrom the NFP integrated circuit. The inverse PCP remap operationreverses the effect of PCP flow merging as well as PCP flow reordering,and ensures that the correct PCP flows are paused in response to thedetecting of a virtual channel overload condition.

Further details and embodiments and techniques are described in thedetailed description below. This summary does not purport to define theinvention. The invention is defined by the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, where like numerals indicate like components,illustrate embodiments of the invention.

FIG. 1 is a diagram of a network device 1 in accordance with one novelaspect.

FIG. 2 is a simplified top-down diagram of the NFP integrated circuit ofthe network device of FIG. 1.

FIG. 3 is a diagram that illustrates a flow of network informationthrough the NFP integrated circuit of FIG. 2.

FIG. 4 is a diagram of the ingress MAC island of the NFP integratedcircuit of FIG. 2.

FIG. 5 is a more detailed diagram of one of the SerDes circuits of theNFP integrated circuit of FIG. 2.

FIG. 6 is diagram of the ingress NBI island of the NFP integratedcircuit of FIG. 2.

FIG. 7 is diagram of the egress NBI island of the NFP integrated circuitof FIG. 2.

FIG. 8 is a diagram of the egress MAC island of the NFP integratedcircuit of FIG. 2.

FIG. 9 is a more detailed diagram of CORE1 of the ingress MAC island.

FIG. 10A is the left half of a larger FIG. 10.

FIG. 10B is the right half of a larger FIG. 10.

FIG. 11 is a diagram of an ethernet frame that has a VLAN tag and a PCPcode field.

FIG. 12A is a diagram that shows how the ingress MAC circuitry of theNFP integrated circuit can be configured into a first set of physicalMAC ports.

FIG. 12B is a diagram that shows how the ingress MAC circuitry of theNFP integrated circuit can be configured into a second set of physicalMAC ports.

FIG. 12C is a diagram that shows how the ingress MAC circuitry of theNFP integrated circuit can be configured into a third set of physicalMAC ports.

FIG. 12D is a diagram that shows how the ingress MAC circuitry of theNFP integrated circuit can be configured into a fourth set of physicalMAC ports.

FIG. 13 is a diagram that illustrates PCP flow merging in accordancewith a first novel aspect, and that illustrates PCP flow reordering inaccordance with a second novel aspect.

FIG. 14 is a diagram that illustrates the contents of the PCP remap LUTcircuit of the first port enqueue engine that handles the first physicalMAC port, for the example of FIG. 13.

FIG. 15 is a diagram that illustrates the contents of the PCP remap LUTcircuit of the second port enqueue engine that handles the secondphysical MAC port, for the example of FIG. 13.

FIG. 16 is a diagram of a dual linked list system in accordance with athird novel aspect.

FIG. 17 is a diagram that illustrates the first and second linked liststhat together comprise a dual linked list.

FIG. 18 is a diagram that illustrates how a dequeue request is handledby the link manager and the pipelined linked list memory.

FIG. 19 is a simplified waveform diagram that illustrates how two rapiddequeue requests (of the dual linked list for the same virtual channel)are handled by the link manager and the pipelined linked list memory.

FIG. 20A is a part of a larger diagram (FIG. 20) that illustrates apushing and a popping of another embodiment of one of the two linkedlists of a novel dual linked list.

FIG. 20B is a part of a larger diagram (FIG. 20) that illustrates apushing and a popping of another embodiment of one of the two linkedlists of a novel dual linked list.

FIG. 20C is a part of a larger diagram (FIG. 20) that illustrates apushing and a popping of another embodiment of one of the two linkedlists of a novel dual linked list.

FIG. 20D is a part of a larger diagram (FIG. 20) that illustrates apushing and a popping of another embodiment of one of the two linkedlists of a novel dual linked list.

FIG. 20E is a part of a larger diagram (FIG. 20) that illustrates apushing and a popping of another embodiment of one of the two linkedlists of a novel dual linked list.

FIG. 21 is a diagram of a PFC pause frame that is generated as a resultof inverse PCP remapping in an overload condition in accordance with afourth novel aspect.

FIG. 22 is a flowchart of a method that involves inverse PCP flowremapping and generation of PFC pause frame in accordance with thefourth novel aspect.

DETAILED DESCRIPTION

Reference will now be made in detail to background examples and someembodiments of the invention, examples of which are illustrated in theaccompanying drawings.

FIG. 1 is a high-level block diagram of a network device 1 in accordancewith one novel aspect. The network device 1 includes and management card2 and multiple line cards 3, 4 and 5, that are coupled to a backplane23. The line cards are of identical construction. Each line card canreceive 120 Gbps (gigabits per second) packet traffic via fiber opticcable 7 and can also transmit 120 Gbps packet traffic out of fiber opticcable 8. Each line card can also receive 100 Gbps packet traffic fromthe switch fabric 6 and can also transmit 100 Gbps packet traffic to theswitch fabric 6. Line card 3 includes, among other parts notillustrated, a Network Flow Processor (NFP) integrated circuit 9, afirst optical transceiver 10, a first PHY integrated circuit 11, asecond optical transceiver 12, a second PHY integrated circuit 13, aconfiguration Programmable Read Only Memory (PROM) 14, and an amount ofexternal Dynamic Random Access Memory (DRAM) 15-20. Packet data receivedfrom a network via optical cable 7 is converted into electrical signalsby first optical transceiver 10. First PHY integrated circuit 11receives the packet data in electrical form and forwards the packet datato the NFP integrated circuit 9 via SerDes connections 21. In oneexample, the packets are directed out of the line card 3 to opticalfiber 8 via SerDes connections 22, second PHY integrated circuit 13, andthe second optical transceiver 12. Alternatively, the packets aredirected from the NFP integrated circuit 9 to the switch fabric 6 viaSerDes connections 24. Packet data from the switch fabric 6 can also becommunicated from the switch fabric 6, across SerDes connections 25, andto the NFP integrated circuit 9. In one example, this packet data isdirected to pass out of the NFP integrated circuit 9 and to opticalfiber 8. In another example, this packet data is directed to pass out ofthe NFP integrated circuit 9 and back to the switch fabric 6 via SerDesconnections 24. Data passing through the NFP integrated circuit 9 may bebuffered in the DRAM 15-20.

FIG. 2 is a top-down diagram of the NFP integrated circuit 9 of FIG. 1.The NFP integrated circuit 9 includes a peripheral first area ofinput/output circuit blocks 26-50. SerDes circuit blocks 26-31 areusable to communicate with optical fibers 7 and 8. Each of these SerDescircuits is duplex in that it has four 10 Gbps lanes for receivingSerDes information and it also has four 10 Gbps lanes for transmittingSerDes information. A SerDes circuit can communicate information in bothdirections simultaneously. Respective ones of the DDR physicalinterfaces 34, 35, 37, 38, 49 and 50 are used to communicate withcorresponding external memory integrated circuits 15-20, respectively.GPIO interface block 36 is used to receive configuration informationfrom external PROM 14.

In addition to the first peripheral area of I/O blocks, the NFPintegrated circuit 9 also includes a second tiling area of islands51-75. Each of the islands 51-75 is either a full rectangular shape, oris half the size of the full rectangular shape. For example, the island67 is a full island. The island 69 is a half island. The functionalcircuits in the various islands of this second tiling area areinterconnected by: 1) a configurable mesh Command/Push/Pull (CPP) databus, 2) a configurable mesh control bus, and 3) a configurable meshevent bus. Each such mesh bus extends over the two-dimensional space ofislands with a regular grid or “mesh” pattern.

In addition to the second tiling area, there is a third area of largersized blocks 76-80. The mesh bus structures do not extend into or overany of these larger blocks. The functional circuitry of a larger sizedblock may connect by direct dedicated connections to an interface islandwithin the tiling area and through this interface island achieveconnectivity to the mesh buses and other islands.

FIG. 3 is a diagram that illustrates one example of packet trafficpassing through the NFP integrated circuit 9 of FIG. 1. Packet trafficis received onto the line card 3 from optical fiber 7, and passesthrough optics transceiver 10, and PHY integrated circuit 11, and acrossSerDes connections 21 into three SerDes circuit blocks 29, 30 and 31.Each of the three SerDes circuit blocks has four 10 Gbps input lanes, sooverall the SerDes circuit block can receive incoming packet data at arate of 40 Gbps. The incoming packet data is spread over three suchSerDes circuit blocks, so the NFP integrated circuit 9 can receive 120Gbps packet data from optical fiber 7. The packet data in thisparticular example passes through dedicated connections from threeSerDes circuit blocks 29-31 to the ingress MAC island 72. The dashedline 81 in FIG. 2 indicates the six SerDes circuit blocks that arecoupled by dedicated connections to the ingress MAC island 72. IngressMAC island 72 converts successive symbols delivered by the physicalcoding layer into packets by mapping symbols to octets, by performingpacket framing, and then by buffering the resulting packets in an SRAMmemory for subsequent communication to other processing circuitry. Afterbuffering in the SRAM, the resulting packets are communicated fromingress MAC island 72 across a single private inter-island minipacketbus, to ingress NBI (Network Bus Interface) island 73. Prepended to thebeginning of each packet is a MAC prepend value that containsinformation about the packet and results of analyses (parse results PR)performed by the ingress MAC island.

For each packet, the functional circuitry of ingress NBI island 73examines fields in the header portion to determine what storage strategyto use to place the packet into memory. In one example, the ingress NBIisland examines the header portion and from that determines whether thepacket is an exception packet or whether the packet is a fast-pathpacket. If the packet is an exception packet, then the ingress NBIisland 73 determines a first storage strategy to be used to store thepacket so that relatively involved exception processing can be performedefficiently, whereas if the packet is a fast-path packet then theingress NBI island determines a second storage strategy to be used tostore the packet for more efficient transmission of the packet from theNFP integrated circuit 9. The ingress NBI island examines a packetheader, performs packet preclassification, determines that the packet isa fast-path packet, and determines that the header portion of the packetshould passes to ME (Microengine) island 67. The header portion of thepacket is therefore communicated across the configurable mesh data busfrom ingress NBI island 73 to ME island 67. The ME island 67 determinesheader modification and queuing strategy for the packet based on thepacket flow (derived from packet header and contents) and the ME island67 informs a second NBI island 64 (also referred to as an egress NBIisland) of these. In this simplified example being described, thepayload portions of fast-path packets are placed into internal SRAM(Static Random Access Memory) MU block 79 and the payload portions ofexception packets are placed into external DRAM 19 and 20. Half island68 is an interface island through which all information passing into,and out of, SRAM MU block 79 passes. The functional circuitry withinhalf island 68 serves as the interface and control circuitry for theSRAM within block 79. Accordingly, the payload portion of the incomingfast-path packet is communicated from ingress NBI island 73, across theconfigurable mesh data bus to SRAM control island 68, and from controlisland 68, to the interface circuitry in block 79, and to the internalSRAM circuitry of block 79. The internal SRAM of block 79 stores thepayloads so that they can be accessed for flow determination by the MEisland 67.

In addition, a preclassifier in the ingress NBI island 73 determinesthat the payload portions for others of the packets should be stored inexternal DRAM 19 and 20. For example, the payload portions for exceptionpackets are stored in external DRAM 19 and 20. Interface island 71, IPblock 80, and DDR PHY I/O blocks 49 and 50 serve as the interface andcontrol for external DRAM integrated circuits 19 and 20. The payloadportions of the exception packets are therefore communicated across theconfigurable mesh data bus from ingress NBI island 73, to interface andcontrol island 71, to external MU SRAM block 80, to 32-bit DDR PHY I/Oblocks 49 and 50, and to external DRAM integrated circuits 19 and 20. Atthis point in the operational example, the packet header portions andtheir associated payload portions are stored in different places. Thepayload portions of fast-path packets are stored in internal SRAM in MUblock 79, whereas the payload portions of exception packets are storedin external memories 19 and 20.

ME island 67 informs egress NBI island 64 where the packet headers andthe packet payloads can be found and provides the egress NBI island 64with an egress packet descriptor for each packet. The egress packetdescriptor indicates a queuing strategy to be used on the packet. EgressNBI island 64 uses the egress packet descriptor to read the packetheaders and any header modification from ME island 667 and to read thepacket payloads from either internal SRAM 79 or external DRAMs 19 and20. Egress NBI island 64 places packet descriptors for packets to beoutput into the correct order. For each packet that is then scheduled tobe transmitted, the egress NBI island uses the packet descriptor to readthe header portion and any header modification and the payload portionand to assemble the packet to be transmitted. The egress NBI island thenperforms packet modification on the packet, and the resulting modifiedpacket then passes from egress NBI island 64 and to egress MAC island65.

Egress MAC island 65 buffers the packets, and converts them intosymbols. The symbols are then delivered by dedicated conductors from theMAC island 65 to three SerDes circuits 42-44. The dashed line 82 in FIG.2 indicates the six SerDes circuits that are coupled by dedicatedconnections to the egress MAC island 65. Although the SerDes circuits42-44 together can provide 120 Gbps of communication throughput, thethroughput out of NFP integrated circuit 9 to the switch fabric 6 islimited to 100 Gbps by the switch fabric. From SerDes circuits 42-44,the 100 Gbps outgoing packets pass out of the NFP integrated circuit 9and across SerDes connections 24 and to switch fabric 6.

The term “packet” is used in the description above in a somewhat looseand colloquial way as is common in the art. More particularly, incomingsymbols are converted into MAC layer “frames”, such as ethernet frames,and a MAC prepend value is prepended to each such frame. The framegenerally carries a single higher level “packet”, such as an IP packet,and that packet is not segmented. A single packet is not being carriedby multiple frames. Accordingly, throughout the rest of this patentdocument the term “frame” and “packet” are used interchangeably to referto MAC layer frames, unless the use of the term “packet” in contextmakes it clear that a different usable of the term is being employed.

FIG. 4 is a more detailed diagram of SerDes circuit blocks 26-31 andingress MAC island 72.

FIG. 5 is a more detailed diagram of one of the SerDes circuit blocks,SerDes circuit block 31. All the SerDes circuit blocks are identical.SerDes circuit block 31 has four 20-bit outgoing PMA (Physical MediumAttachment) buses 83-86, and four 20-bit incoming PMA buses 87-90.

The SerDes circuit bocks of FIG. 4 are being used to receive frame data,so the outgoing 20-bit PMA buses are not illustrated in FIG. 4. PMA RXdata 91 is converted into frames by MAC layer interface circuit block92. Blocks 92 and 93 are identical. Each of these blocks actuallyincludes an Interlaken portion and an ethernet portion. MAC layerinterface circuit block 92 analyzes the frame data and places theresults at the beginning of the frame data in the form of the “MACprepend” value. The resulting frames and their associated MAC prependvalues are then buffered in SRAM 94. Reference numeral 95 identifies apart of a block that represents one frame and reference numeral 96identifies a part of the block that represents the MAC prepend value.The frame 95 and its MAC prepend value 96 are not stored this way. Theillustration of the blocks 95 and 96 is provided for instructionpurposes. The MAC prepend value includes: 1) an indication of the lengthof the frame (packet), 2) an indication whether the frame contains an IPpacket, 3) and indication of whether the checksums are correct, and 4) atime stamp indicating when the packet was received.

As such frames are loaded into SRAM 94, a statistics block 97 counts thenumber of frames that meet certain criteria. Various sub-circuits of theingress MAC island 72 are configurable. The input conductors 98 labeledCB couples the certain portions of the ingress MAC island to the controlmesh bus (CB) so that these portions receive configuration informationfrom the root of control bus tree and configuration PROM 14. SRAM block94 includes error detection and correction circuitry (ECC) 99. Errorinformation detected and collected by ECC block 99 and statistics block97 is reported through a local event bus and a global event chain backto the ARM island 52 by the event bus mesh. Ingress MAC island 72 iscoupled to part of one of the local event rings. Event packets arecirculated into the MAC island via conductors 100 and are circulated outof the MAC island via conductors 101.

Frames that are buffered in SRAM 94 are then output from the ingress MACisland 72 to the ingress NBI island 73 in the form of one or more256-byte minipacket transactions 102 communicated across dedicatedminipacket bus connections 103. The minipacket bus is a private bus usedto transport this frame information from the MAC island to the NBIisland. Each minipacket bus transaction consists of: 1) 256 bytes offrame data, or the remaining amount of data for the frame, whichever isless, and the MAC prepend. The MAC prepend includes: 1) a virtualchannel number to which this frame data belongs, 2) a SOF (Start ofFrame) bit indicating if the 256 bytes of frame data is the first pieceof a frame for this virtual channel, 3) an EOF (End of Frame) bit thatindicates whether the 256 bytes of frame data is the last piece of aframe for this virtual channel. Statistics information 104 is alsocommunicated to the ingress NBI island 73 via a separate XPB bus 105.

FIG. 6 is a diagram of ingress NBI island 73. Ingress NBI island 73receives the MAC prepend and the minipacket information via dedicatedminipacket bus connections 103 from the ingress MAC island 72. The first256 bytes of the frame and the MAC prepend pass through multiplexingcircuitry and are analyzed by a pool 106 of forty-eight picoengines.Pool 106 generates preclassification results 107. The preclassificationresults 107 include: 1) a determination of which one of multiple bufferpools to use to store the frame, 2) a sequence number for the frame in aparticular flow of frames through the NFP integrated circuit, and 3)user metadata. The user metadata is typically a code generated by thepicoengine pool 106, where the code communicates certain informationabout the packet. In one example, the user metadata includes a bit thatindicates whether the frame was determined by the picoengine pool 106 tobe a first type of frame (an exception frame or packet), or whether theframe was determined to contain a second type of frame (a fast-pathframe or packet). The frame is buffered in SRAM 108. A buffer pool is aset of targets in ME islands where header portions can be placed. Abuffer list is a list of memory addresses where payload portions can beplaced. DMA engine 109 can read the frame out of SRAM 108 via conductors110, then use the buffer pools to determine a destination to which theframe header is to be DMA transferred, and use the buffer lists todetermine a destination to which the frame payload is to be DMAtransferred. The DMA transfers occur across the configurable mesh databus. In the case of an exception packet, the preclassification usermetadata and buffer pool number indicate to the DMA engine 109 that theframe is an exception frame and this causes a first buffer pool and afirst different buffer list to be used, whereas in the case of afast-path frame the preclassification user metadata and buffer poolnumber indicate to the DMA engine that the frame is a fast-path frameand this causes a second buffer pool and a second buffer list to beused. CPP bus interface 111 is a CPP bus target. CPP bus interface 111is CPP bus interface through which the configurable mesh data bus inaccessed. Arrow 112 represents frames (packets) that are DMA transferredout of the ingress NBI island 73 by DMA engine 109 and through CCP businterface 111. Each frame (packet) is output with a correspondingingress packet descriptor. An ingress packet descriptor includes: 1) anaddress indicating where and in which ME island the header portion isstored, 2) an address indicating where and in which MU island thepayload portion is, 3) how long the frame is, 4) a sequence number forthe flow to which the frame belongs, 5) user metadata.

Configuration data can be written by a master on the CPP data bus (DB),through DB interface 111, through CPP-to-XPB bus interface 113, acrossthe XPB bus 105, and into the ingress MAC island 72. Arrows 114 in FIG.6 and in FIG. 4 represent this configuration information. As describedin further detail below, this configuration information 114 configuresthe MAC layer interface circuit blocks 92 and 93 so that the blocks 92and 93 together with the SerDes circuits 26-31 are partitioned into andconfigured into a configurable number of “physical MAC ports”. Also, asdescribed in further detail below, this configuration information 114also: 1) configures block 94 to support a configurable number of“virtual channels”, 2) configures how block 94 performs PCP mapping, 3)configures how block 94 performs PCP flow merging, 4) configures howblock 94 performs PCP flow reordering, 5) configures how block 94performs inverse PCP remapping for PFC pause frame generation.

After the picoengine pool 106 in the ingress NBI island 73 has done itsanalysis and generated its preclassification results for the packet, theingress NBI island then DMA transfers the frame headers (packet headers)and associated preclassification results across the CPP configurablemesh data bus DB and into the ME island 67. Within the ME island 67, oneor more microengines then performs further processing on the header andpreclassification results as explained in further detail in U.S. patentapplication Ser. No. 13/399,888, entitled “Island-Based Network FlowProcessor Integrated Circuit”, filed Feb. 17, 2012, by Stark et al. (theentire subject matter of which is hereby incorporated by reference).

FIG. 7 is a diagram of egress NBI island 64. As illustrated by thearrows of FIG. 3, header and control information passes from the MEisland 67 to the NBI island 64, and packet data passes from where it isstored (either SRAM 52 or external DRAM 19-20) to the egress NBI island64. From the egress NBI island, packet data passes across a minipacketbus 115 to the egress MAC island 65.

FIG. 8 is a diagram of egress MAC island 65. In the presently describedexample, the packet traffic discussed in connection with FIG. 3 flowsout of the egress MAC island 65 and through three SerDes circuits 42-44to the switch fabric 6. Although the ingress MAC island 72 isillustrated in FIG. 4 as having only ingress circuitry, and although theegress MAC island 65 is illustrated in FIG. 8 as having only egresscircuitry, both MAC islands are actually identical and each of the twoislands has both ingress and egress circuitry. Ingress MAC island 72 canoutput packet traffic via the same three SerDes circuits 29-31 tooptical cable 8. Egress MAC island 65 can receive packet traffic via thesame three SerDes circuits 42-44 from switch fabric 6.

FIG. 9 is a more detailed block diagram of the ingress MAC island 72.Ingress MAC island 72 includes two cores, referred to here as CORE1 andas CORE2, and a DWRR (Deficit Weighted Round Robin) arbiter andminipacket bus interface 116. The two cores are structurally identical.As illustrated in FIG. 9, the SRAM block 94 of FIG. 4 actually includesmuch more circuitry than just SRAM circuitry. The SRAM block 94 of FIG.4 actually includes the parser and checksum circuitry of the two cores,the port enqueue circuitry of the two cores, the SRAMs of the two cores,the port dequeue circuitry of the two cores, and the link managercircuit of the two cores, along with the common DWRR arbiter andminipacket bus interface 116. The two parser and checksum circuits ofCORE1 are identified in FIG. 9 by reference numerals 117 and 118. Theport enqueue circuitry of CORE1 is identified in FIG. 9 by referencenumeral 119. The SRAM of CORE1 is identified in FIG. 9 by referencenumeral 120. The port dequeue circuitry of CORE1 is identified in FIG. 9by reference numeral 121. The link manager circuit of CORE1 isidentified in FIG. 9 by reference numeral 122. Three of the six SerDescircuits that work with the ingress MAC island are coupled to CORE1,whereas the other three are coupled to CORE2. MAC layer interfacecircuit block 92 has an Ethernet MAC portion 123 and an InterLaken MACportion 124. The Ethernet MAC portion 123 of block 92, in one example,is a commercially available IP core of the “Hydra” family, referred toas “Multi-Channel/Multi-Rate 12 Lane 1/10/40/100 G Ethernet MAC/PCSCore”, ordering code: MTIP-H12LANE1040100-lang-tech, available fromMorethanIP GmbH, Muenchner Strasse 199, D-85757 Karlsfeld, Germany.

Based on configuration information 114, the Ethernet MAC portion 123,along with SerDes circuits 29-31, is configured into a desired number of“physical MAC ports”. The Ethernet MAC portion 123 includes aconfiguration register 125 that is loaded with configuration information114 for this purpose. Translation circuit 182 translates XPB buscommunications into communications understood by the Ethernet MACportion 123. The port enqueue circuitry 119 includes thirteen portenqueue engines. The port enqueue engines are labeled one throughthirteen in the diagram of FIG. 9. The configuration register 126 of theport enqueue circuitry 119 is loaded with configuration information 114such that one port enqueue engine is assigned to each of the physicalMAC ports. Likewise, the port dequeue circuitry 121 includes thirteenport dequeue engines. The port dequeue engines are labeled one throughthirteen in the diagram of FIG. 9. The configuration register 127 of theport dequeue circuitry 121 is loaded with configuration information 114such that one port dequeue engine is assigned to each of the physicalMAC ports.

In one example, ethernet frames are received on each of the physical MACports. Frame data of such an ethernet frame is output, 256 bits at atime, onto TDM (Time Division Multiplexed) bus 129. Each such 256-bitamount of packet data is accompanied by: 1) a value that indicates thephysical MAC port that received the packet data, 2) a SOF (Start ofFrame) bit that if asserted indicates that the 256-bit amount of packetdata carries the first packet data of a frame, 3) an EOF (End of Frame)bit that if asserted indicates that the 256-bit amount of packet datacarries the last packet data of a frame, 4) an error bit ERR, 5) a 5-bitMOD value that is valid if EOF is asserted and in that case indicateshow many bytes of the 256-bit value are valid, 6) a port number, and 7)a timestamp that is valid if SOF is asserted. This additionalinformation about the 256-bit amount of packet data is generated by theEthernet MAC portion 123 of the MAC layer interface circuit 92. These256-bit values along with their accompanying descriptive information aresupplied one after another, in time division multiplexed fashion, fromthe various physical MAC ports onto TDM bus 129.

A 256-bit value is supplied to parser and checksum circuit 117, and isalso supplied to the port enqueue circuitry 119. One of the port enqueueengines of the port enqueue circuitry 119 is hardcoded with the numberof the physical MAC port. Each such port enqueue engine receives thephysical MAC number and determines, using its hardcoded number, if the256-bit value is for the port handled by the port enqueue engine. Theproper port enqueue engine (the one whose hardcoded number matches theport number of the incoming 256-bit value) receives the 256-bit value,and loads the value into a buffer for the appropriate one of virtualchannels. The buffer is in SRAM 120. Eight such 256-bit writes arerequired to fill the buffer. The port enqueue engine operatesatomically, one frame at a time, loading buffers with frame data fromSOF to EOF, to a single channel. The Ethernet MAC portion 123 (the“Hydra”) presents 256-bit frame data for each port atomically. Framedata for multiple ports may be interleaved on the TDM bus (e.g., Port 1SOF, Port 2 SOF, . . . , Port 1 EOF, Port 2 EOF), but each enqueueengine only takes the data for its assigned port, so each enqueue enginereads frames atomically. At the time of loading the last 256-bit word ofa frame, the parser and checksum circuit 117 has finished generating the“parser result” (PR) value. The PR value is then into a “PD and PRMemory” 131 in the SRAM 120, where the result value (PR) written isstored so that it is indexed by the buffer ID of the first buffer thatstores the first 256-bit value of the frame. In addition to the parseresult (PR) value, the timestamp value is also written into this “PD andPR Memory” 131, indexed to the buffer ID of the first buffer that storesthe first 256-bit value of the frame.

When the last 256-bit value has been written into a buffer such that thebuffer is full, a buffer ID for the buffer is added to a linked list forthe virtual channel. This linked list stores the buffer IDs of thebuffers that store the frame data. As buffer IDs are stored into thelinked list, the associated frame data is said to be pushed into thelinked list of buffers. In the illustration of FIG. 9, there aresixty-four linked lists of buffers, where there is one linked list ofbuffers for each of a maximum sixty-four corresponding virtual channels.The ingress MAC island is configured to provide one linked list ofbuffers for each virtual channel through the NFP integrated circuit fromthe optical fiber 7 to the switch fabric 6 in this example. The ingressMAC island supports a maximum of 128 virtual channels, with each coresupporting a different set of sixty-four virtual channels. A framereceived via a physical MAC port can have an IEEE 802.1Q PCP (PriorityCode Point) value from zero to seven. The frames received on aparticular physical MAC port having a particular PCP value are referredto here as a “PCP flow”. Each of the eight possible PCP flows receivedonto a given physical MAC port can be assigned to a different one ofeight linked lists of buffers for eight corresponding virtual channels,such that there is a one-to-one correspondence between PCP flows andvirtual channels. Alternatively, multiple ones of the PCP flows cominginto a given physical MAC port can be merged so that the frames of thesemerged flows are all stored into the same linked list of buffers for thesame one virtual channel. Even though PCP flows may be merged and storedin the same virtual linked list of buffers, the buffers for frames arestored (in the linked list of buffers for the virtual channel) so thatframes are stored atomically, one after the other. The link manager 122handles head pointer and tail pointer maintenance for the linked lists.The link manager 122 also manages a link list of buffers, where thebuffers are free buffers. For each virtual channel, the link manager 122also maintains a buffer count and a packet count. The buffer countindicates the number of buffers that are in the linked list of buffersfor the virtual channel. The packet count indicates the number ofcomplete packets (i.e., frames) that are stored in the linked list ofbuffers for the virtual channel. A frame (or packet) that is stored insuch a linked list of buffers in SRAM 120 is said to be buffered in theSRAM.

The port dequeue circuitry 121 dequeues the linked lists of buffers forthe virtual channels and supplies the frame data, one buffer at a time,to the DWRR arbiter and minipacket bus interface 116. The DWRR arbiterand minipacket bus interface 116 in turn outputs 256-byte minipackets,out of the ingress MAC island 72 via minipacket bus 103.

Each port dequeue port engine examines the frame counts for the virtualchannels that are receiving frames from the physical MAC port associatedwith the port dequeue engine. There is one port dequeue engine for eachphysical MAC port, which handles all the channels for that physical MACport. Accordingly, the port dequeue engine for the first physical MACport examines the frame counts for the linked lists that store framedata for PCP flows received onto the first physical MAC port. Thevarious port dequeue engines that determine that their linked lists haveat least one complete frame to dequeue then arbitrate with the DWRRarbiter and minipacket bus interface 116 to read the SRAM 120, and oneis granted permission to dequeue buffers. If, for that selected portdequeue engine, there are linked lists for more than one virtual channelthat are indicated (by their frame counts) to have at least one framestored, then a local arbiter within the selected port dequeue engineselects one of the virtual channels to be dequeued. The selected portdequeue engine then dequeues buffers for one frame (dequeues this frameatomically) from the SRAM frame memory for the selected virtual channel.At the time the first buffer of a frame is dequeued, the packetdescriptor (PD) (for that frame) is also automatically read by the portdequeue engine via the link manager. The term “packet descriptor” isused here, even though the descriptor is perhaps more accuratelyreferred to as a “frame descriptor”. The packet descriptor (PD) wasstored indexed to the buffer ID of the first buffer that stores thefirst part of the frame, so the packet descriptor (PD) can be read frommemory at this time. The packet descriptor was generated, and caused tobe stored in the PD and PR memory, by the particular port enqueue enginethat enqueued to associated frame. The packet descriptor (PD) contains anumber that indicates the numbers of buffers that store the frame, andalso contains a number of bytes in the last buffer that marks the end ofthe frame. This packet descriptor (PD) information is generated the portenqueue engine. Together these two values indicate how many 256-bitreads the port dequeue engine must perform on the linked list of buffersuntil one entire frame has been atomically read. After reading the first256-bit value and the packet descriptor, the port dequeue engine thengoes on to read subsequent 256-bit values, one by one, from subsequentbuffers of the same linked list of buffers, and these 256-bit values aresupplied to the DWRR arbiter and minipacket bus interface 116 to readthe SRAM 120.

As indicated above, the DWRR arbiter and minipacket bus interface 116outputs the data in 256-byte minipackets. When the last valid word of abuffer has been read, then the DWRR arbiter 116 arbitrates again andselects another port dequeue engine. The DWRR arbiter and minipacket businterface 116 is a 26-slot arbiter with programmable weights. Theweights are in bytes. When a port is arbitrating and is credit-positiveand wins a grant, then the arbiter deducts 512 bytes from the port'scredits. The port can then output up to 512 bytes. Once the buffertransfer ends and the remaining transfer credits of the 512 bytes isless than 512 bytes, the port dequeue engine will cause the arbiter'sport credit count to be updated with the unused remaining credits of theoriginal 512 bytes. When no port dequeue engines are arbitrating, orwhen all the port dequeue engines have negative credit, the arbitercredit counts are refreshed to the original allotment. By programmingall the ports to an equal initial weight, the dequeue bandwidth isevenly distributed across the arbitrating ports.

FIG. 10A and FIG. 10B together form a single larger FIG. 10. FIG. 10 isa more detailed diagram of the port enqueue circuitry 119, the SRAM 120,the port dequeue circuitry 121, and the link manager 122 of FIG. 9. SRAM120 actually is a collection of multiple separately accessible memories:the buffer memory 128, the linked list memory 129, the free bufferlinked list memory 130, and the PD (Packet Descriptor) and PR (ParseResult) memory 131. Each buffer in buffer memory 128 is eight words, of256 bits for each word. The buffer memory 128 is read and written one256-bit word at a time. Any buffers that are not being used to storeframe data are referred to as “free”. A pointer to each of these freebuffers is maintained in a “free buffer linked list” in the free bufferlinked list memory 130. The link manager 122 can pop the free bufferlinked list, thereby obtaining a buffer ID that points to a free bufferin the buffer memory 128. This buffer ID can then be used to write framedata into the corresponding buffer that was free. When data is writteninto the buffer, the buffer is no longer “free”. When the buffer hasbeen filled, its buffer ID is pushed onto the appropriate one of thelinked lists in linked list memory 129. For each virtual channel, thereis one linked list of buffers, and the buffer IDs of these buffers arestored in a corresponding linked list in the linked list memory. In anenqueue operation, when frame data is to be written into the linked listof buffers for a particular virtual channel, the link manager 122 canpush the buffer ID of the now-used buffer onto the linked list for thevirtual channel. The link manager 122 pops the free buffer list, therebyobtaining a buffer ID for another buffer to be used to store the nextamount of frame data. In a dequeueing operation, the link manager 122can pop the appropriate linked list, and obtain the next buffer ID ofthe next buffer to be read (for a given virtual channel). Once the framedata has been read out of that buffer, the buffer ID for that buffer ispushed onto the free buffer linked list. The link manager 122 handlesthe pushing and popping from the various linked lists, and does so inresponse to requests and commands from the port enqueue engines and portdequeue engines.

As described above, for each complete frame stored in a linked list ofbuffers, the “PD and PR memory” 131 stores a packet descriptor (PD)value and a parse result (PR) value, where these PD and PR values arestored indexed to the buffer ID of the buffer that stores the first partof the packet. Upon enqueueing the buffer that stores the last frame(EOF) data for a frame, the packet descriptor (PD) and parse result (PR)is written into the PD and PR memory 131. Each of the port enqueueengines has a set of PD output conductors that are coupled to inputs ofan OR gate structure, where the output of the OR gate structure issupplied to the PD and PR memory. Because only one of the port enqueueengines can drive a non-zero PD value at a given time, the active portenqueue engine outputs the packet descriptor (PD) that passes throughthe OR gate structure. The supplying of the packet descriptor onto theinputs of the PD and PR memory results in writing of the PD into thememory. Upon dequeueing the buffer that stores the first frame data of aframe, the packet descriptor (PD) is read from the PD and PR memory 131and is supplied to all the port dequeue engines at the same time inparallel. Only the appropriate active port dequeue engine, however,latches in the packet descriptor.

How a 256-bit value from the TDM bus 129 is processed by a port enqueueengine 133 and a port dequeue engine 134 is now described in referenceto FIG. 10. The 256-bit value 132 is received on conductors 129, alongwith a port number, an error bit ERR, five MOD bits (if EOF is assertedthen these bits are valid and indicate how many bytes are valid in this256-bit value), a timestamp, an SOF bit (if this bit is asserted thenthis 256-bit value contains the first part of the frame), and an EOF bit(if this bit is asserted then this 256-bit value contains the last partof the frame).

The parse and checksum circuit 117 begins analyzing such 256-bit valuesfor a frame, and when the EOF 256-bit value has been received andprocessed, the parse and checksum circuit 117 supplies the parse resultto the PD and PR memory 131. The parse result PR is written into the PDand PR memory 131 at that time.

Port enqueue engine 133 has circuitry 135 that is hardcoded with a portnumber. In the presently described example, the circuit 136 uses thishardcoded value to determine whether the port number of the incoming256-bit value matches the hardcoded value. For a given 256-bit value, ifthere is a match the port enqueue engine 133 processes the 256-bitvalue, otherwise the port enqueue engine 133 takes no action. Wherethere is a match, circuit 137 extracts the PCP value 139 from the MACheader of the frame. The 3-bit PCP value 139 is supplied to PCP RemapLookup Table (PRLUT) circuit 138, that in turn outputs a remapped 3-bitvalue 140. The PRLUT circuit 138 is a LUT that has eight three-bitentries, one entry for each 3-bit PCP value. The configurable contentsof the PRLUT circuit 138 is provided by configuration register 141.PRLUT circuit 138 is a set of 3-bit loadable registers and an associatedset of three 8:1 output multiplexers, where the select inputs of theoutput multiplexers are controlled by the 3-bit PCP value, and where thedata outputs of the multiplexers output the 3-bit remapped PCP value140. The remapped PCP value 140 is then added by an adder 143 to a 6-bitbase value supplied by a base register 142. The resulting 6-bit sum is avirtual channel number. The PCP remapping circuitry 137, 138, 143, 141and 142 is preconfigured high-speed combinatorial logic circuitry thatgenerates virtual channel numbers and that includes no processor thatfetches or executes any instruction.

If a SOF is indicated by SOF detector 145, then circuit 136 issues arequest for a buffer ID from the link manager 122. This request iscommunicated via conductors 146, and through an OR gate structure, tolink manager 122. Only one port enqueue engine can output a non-zerorequest (request for a buffer ID) at a time, so the request passesthrough the OR gate structure and to the link manager. In response, thelink manager 122 pops the head of the free buffer linked list, therebyobtaining a buffer ID of a free buffer, and forwards the buffer ID tothe port enqueue engine 133 via conductors 147. The conductors 147extend to each of the port enqueue engines, but only the active portenqueue engine latches in the buffer ID value. The active port enqueueengine 133 thereafter uses the buffer ID to write into SRAM 120. Thisbuffer ID identifies the beginning of an 8-word by 256-bit buffer. Aseach successive 256-bit word is written, the least significant threebits of the value ADRS[13:0] are incremented due to the incrementing of3-bit counter 148. For each such 256-bit value received onto the portenqueue engine 133, the 3-bit counter 148 increments, and thisincrementing results in the next 256-bit value being written into thenext word of the buffer. Only the port enqueue engine that is active canoutput a non-zero address value. The OR gate structures 149 and 150therefore pass the non-zero address value from the one active portenqueue engine onto the ADRS[13:0] address lines of the SRAM memory.When a buffer has been filled, then the port enqueue engine 133 issues apush buffer command via conductors 151 to the link manager 122 alongwith a virtual channel number. The link manager 122 handles pushing thebuffer ID onto the linked list for the indicated virtual channel. Foreach linked list, the head pointer queue element and the tail pointerqueue element of the linked list are present in the link manager 122,whereas any additional queue elements of the linked list are stored inlinked list memory 129. For the free buffer linked list, the additionalqueue elements are stored in the free buffer linked list memory 130. Asbuffers are filled and as buffer IDs are pushed onto a linked list for avirtual channel, a buffer count for the link list is incremented. Thebuffer count values are labeled “B#” in FIG. 10. Similarly, when thelast 256-bit value of a frame has been loaded into the buffer memory,then a frame count value for the linked list is incremented. The framecount values are labeled “F#” in FIG. 10.

When EOF is asserted (indicating the last write to the last bufferstoring the frame), and the last buffer ID of the frame has been pushedonto the appropriate linked list for a virtual channel, then the portenqueue engine 133 causes the link manager 122 to write a packetdescriptor (PD) into the “PD and PR memory” 131 by issuing a push packetcommand to the link manager 122 via conductors 152 and an OR gatestructure. Again, only one port enqueue engine can output a non-zeropush packet command, the push packet command output by the active portenqueue engine passes through the OR gate structure to the link manager122. The push packet command includes the buffer ID of the first bufferstoring data for the frame, so that the packet descriptor (PD) will bestored indexed to this buffer ID.

After the port enqueue engine 133 receives a 256-bit value, there is afixed number of clock cycles before the ADRS[13:0] address value will bepresented to the SRAM 120. Pipeline registers 184 and 185 are providedto delay the 256-bit data value the appropriate amount so that thecorrect 256-bit data value will be written into SRAM 120. The basenumber BASE stored in base register 142 sets the base virtual channelnumber used by the port enqueue engine. If the port enqueue engine loadsbuffers for multiple virtual channels, then the virtual channel numbersof these virtual channels have incremental offsets from the base virtualchannel number. These virtual channel numbers are a block of virtualchannel numbers, whose base virtual channel number value is set by thecontents of the base register 142.

Those linked lists of buffers, the virtual channels of which areassigned to a physical MAC port, are dequeued by a port dequeue engine.There is one port dequeue engine assigned to handle each physical MACport. In FIG. 10, port dequeue engine 134 is the dequeue engine for samephysical MAC port to which the port enqueue engine 133 is assigned. TheBASE content of base register 153 sets a base virtual channel number,and the NUM content of num register 154 defines a number of virtualchannel numbers in a block of virtual channel numbers. These virtualchannel numbers indicate the virtual channels, the linked lists ofbuffers for which the port dequeue engine is responsible. By writingappropriate BASE and NUM values into the base and num registers, theport dequeue engine is configured to dequeue the corresponding linkedlists of buffers. A 64-bit vector is output from the link manager 122via conductors 155, where each respective bit in the 64-bit vectorindicates whether the frame count (also called packet count) of acorresponding one of the sixty-four linked lists is a non-zero value. Abit being set indicates that the corresponding linked list stores atleast one complete packet (one complete frame). The mask circuit 156 inthe port dequeue engine 134 uses the BASE and NUM values from registers153 and 154 to examine those bits in the 64-bit vector that pertain tothe linked lists that the port dequeue engine 134 is responsible fordequeueing. If more than one of these bits is set, then a local “arbiterand dequeue request generator” 157 selects one of the correspondingvirtual channels. A dequeue request 158 to dequeue the linked list ofbuffers for the selected virtual channel is supplied via an OR gatestructure and conductors 159 to the link manager 122. The dequeuerequest includes a virtual channel number. In response to the dequeuerequest, the link manager returns the buffer ID of the appropriatelinked list. The buffer ID is returned to the port dequeue engine 134via conductors 160. If the indicated buffer is the buffer that storesthe first part of a frame, then the PD and PR memory 131 is read, andthe packet descriptor (PD) and the parse result (PR) are returned to theport dequeue engine via conductors 161. Within the port dequeue engine,the buffer ID value is shifted by three bits by circuit 162. For eachsuccessive read of a word from the buffer, the three-bit counterincrements a count value. The sum of the three-bit counter value and theshifted buffer ID value, as output by adder 164, is supplied through ORstructure 165 as the address value ADRS[13:0] to the SRAM 120. Thisaddress value identifies one of the eight 256-bit words of theappropriate buffer. A few clock cycles later the 256-bit data contentsof the addressed buffer are then output via conductors 166. If the256-bit value is indicated to be the start of frame (SOF), then an SOFbit is asserted by circuit 167. If an EOF circuit 168 determines thatthe 256-bit word is the last of a frame, based on the “number of buffersfor the frame” value and the “bytes in last buffer” value of the packetdescriptor, then the circuit 168 asserts an end of frame (EOF) bit.These bits are determined by circuit 167 several clock cycles before theassociated 256-bit data value is output by SRAM 120, so a MAC prepend,the MOD bits, the SOF bit and the EOF bit are delayed by pipelineregisters 169 and 170. The MAC prepend is derived from the parse result(PR) value received via conductors 161. The value MOD indicates how manybytes are valid in the 256-bit word (in the last word of a frame, onlysome of the bytes are typically valid). Only one of the port dequeueengines is allowed by the DWRR arbiter 116 to dequeue and to outputnon-zero values at a time, so the MAC prepend, MOD, SOF and EOF bitspass through OR structure 171. Prepend and merge circuit 183 “prepends”the MAC prepend value onto the front of the first amount of frame data,but nonetheless the prepend and merge circuit 183 outputs information256 bits at a time. For each 256-bit value that is output, the MOD bits,the SOF bit, the EOF bit, and the virtual channel number as output by ORgate structure 171 are output as sideband signals.

When the last buffer of the frame has been read from SRAM 120, the portdequeue engine 134 releases its request to the DWRR arbiter 166. TheDWRR arbiter 166 is then able to arbitrate again among the various portdequeue engines for which port dequeue engine will be permitted todequeue next. Each port dequeue engine dequeues ethernet framesatomically from the SRAM 120, so once the dequeueing of buffers for aframe has begun, the buffers for that frame continue to be dequeued(when allowed by DWRR arbiter 116) until the last buffer storing datafor that frame has been dequeued.

The link manager 122 stores, for each linked list (for each virtualchannel), an overflow threshold value. The overflow threshold values aredenotes “TV1 through TV64” in FIG. 10. The link manger 122 outputs a64-bit vector signal 172, each bit of which indicates whether the buffercount for a virtual channel is greater than the overflow threshold valuefor that virtual channel. This 64-bit vector signal 172 is received inparallel by all the port enqueue engines. A mask circuit in each portenqueue engine uses the BASE and NUM values to identify which bits fromthe 64-bit vector signal value correspond to virtual channels, thelinked lists of buffers of which that particular port enqueue engine areenqueued by that port enqueue engine. As explained above, each portenqueue engine is configured by BASE and NUM to enqueue up to eightvirtual channels. The circuitry of one port enqueue engine is describedhere as an example. In the case of port enqueue engine 133, the 64-bitvector signal 172 is received by mask circuit 173. Mask circuit 173outputs an indication of all virtual channels that this particular portenqueue engine 133 is responsible for enqueuing. For each such indicatedvirtual channel, an Inverse PCP Remap LUT (IPRLUT) circuit 174 performsinverse PCP mapping, and outputs the PCP value or values associated withthe virtual channel that has exceeded its overflow threshold value.Where multiple PCP flows are merged by PCP remap LUT circuit 138 in theenqueueing process into one virtual channel, that one virtual channelnumber is inverse mapped back to the multiple PCP values of those PCPflows. The effect of PCP merging and/or PCP reodering performed by thePCP LUT circuit 138 in the enqueueing process is therefore reversed inby the IPRLUT circuit 174. For each PCP value for any of the virtualchannels identified by circuits 173 and 174, the corresponding bit in an8-bit priority class enable vector 176 is set. This priority classenable vector 176 is output via conductors 175 to Ethernet MAC portion123. There is one such eight-bit set of conductors that extends fromeach respective port enqueue engine to its corresponding physical MACport portion of the Ethernet MAC portion 123. For a given physical MACport portion, the Ethernet MAC portion 123 uses the incoming priorityclass enable vector 176 (from its corresponding port enqueue engine) togenerate a IEEE 802.3x PFC pause frame. The physical MAC port thatreceived the priority class enable vector 176 then sends out the PFCpause frame. The MAC source address of the PFC pause frame is thedestination MAC address of the physical MAC port. The MAC destinationaddress is a predefined value.

As in the case of the PCP remapping circuitry 137, 138, 143, 141 and 142as described above, the inverse PCP remapping circuitry 173 and 174 isan amount of preconfigured high-speed combinatorial logic circuitry thatthat includes no processor that fetches or executes any instruction.

FIG. 11 is a diagram of an ethernet frame 177 that includes a VLAN tag178 and the 3-bit PCP code 139. The 12-bit VLAN tag and the 3-bit PCPcode are parts of an IEEE 802.1Q header 180. This 3-bit PCP code 139 isthe PCP value 139 identified by circuit 137 in FIG. 10.

FIG. 12A is a diagram that illustrates one possible configuration of theingress MAC circuitry of the NFP integrated circuit 9. The MAC interfacecircuitry and the six SerDes 26-31 are configured into twenty-four 10Gbps physical MAC ports. In this case, the number of physical MAC portsmultiplied by eight (the number of PCP priority levels) exceeds themaximum number of virtual channels supported (sixty-four), so PCP flowmerging (using PCP remap LUT circuit 138) as described above isperformed.

FIG. 12B is a diagram that illustrates another possible configuration ofthe ingress MAC circuitry of the NFP integrated circuit 9. The MACinterface circuitry and the six SerDes 26-31 are configured into six 40Gbps physical MAC ports. The number of physical MAC ports multiplied byeight (the number of PCP priority levels) does not exceed the maximumnumber of virtual channels supported, so PCP flow merging need not beapplied.

FIG. 12C is a diagram that illustrates another possible configuration ofthe ingress MAC circuitry of the NFP integrated circuit 9. The MACinterface circuitry and the six SerDes 26-31 are configured into two 100Gbps physical MAC ports and four 10 Gbps physical MAC ports. The numberof physical MAC ports multiplied by eight (the number of PCP prioritylevels) does not exceed the maximum number of virtual channelssupported, so PCP flow merging need not be applied.

FIG. 12D is a diagram that illustrates another possible configuration ofthe ingress MAC circuitry of the NFP integrated circuit 9. The MACinterface circuitry and the six SerDes 26-31 are configured into twelve10 Gbps physical MAC ports and three 40 Gbps physical MAC ports. Thenumber of physical MAC ports multiplied by eight (the number of PCPpriority levels) exceeds the maximum number of virtual channelssupported, so PCP flow merging is applied.

FIG. 13 is a diagram that illustrates both PCP flow merging, as well asPCP flow reordering. The PCP remap LUT circuit of the port enqueueengine handling the first physical MAC port “1” is configured so thatthe PCP flows of PCP codes 101, 110, and 111 are merged so that alltheir frames are stored into the same linked list of buffers for virtualchannel number 000101. Similarly, the PCP remap LUT circuit of the portenqueue engine handling the third physical MAC port “3” is configured sothat PCP flows for PCP codes 000, 001, 010, 011, 100, 101 and 110 areall merged so that their frames are all stored into the same linked listof buffers for virtual channel number 001101. The first and third portenqueue engines therefore perform PCP flow merging. The second portenqueue engine that is handling the second physical MAC port “2”recorders PCP flows as they are assigned to virtual channels, but doesnot perform flow merging. The PCP remap LUT circuit of the second portenqueue engine assigns the PCP flow of PCP code 011 so that its framesare stored into the linked list of buffers for virtual channel number001010. The PCP remap LUT circuit of the second port enqueue engineassigns the PCP flow of PCP code 100 so that its frames are stored intothe linked list of buffers for virtual channel number 001011. The PCPremap LUT circuit of the second port enqueue engine assigns the PCP flowof PCP code 101 so that its frames are stored into the linked list ofbuffers for virtual channel number 001001. In one example, the largerPCP code values are to indicate higher priority levels, and the higherthe virtual channel number (within those virtual channels assigned tothe same physical MAC port) the higher the priority, but yet the higherpriority PCP flow of PCP code value “101” is assigned to a virtualchannel of lower priority that are the PCP flows for PCP code values“011” and “100”. Higher virtual channel numbers of a physical MAC portgenerally correspond to higher priority, but this need not be so. Therelative priorities of the virtual channels is actually determined bypacket processing functionality downstream of the ingress MAC island.

FIG. 14 is a diagram that illustrates the configuration informationstored in the PCP remap LUT circuit of the first port enqueue engine(the one handling the first physical MAC port), for the example of FIG.13. There are eight 3-bit words stored. The contents of the eight tablelocations of the PCP remap LUT are the values in the right column. The3-bit address values that point to those table location are indicated inthe left column.

FIG. 15 is a diagram that illustrates the configuration informationstored in the PCP remap LUT circuit of the second port enqueue engine(the one handling the second physical MAC port), for the example of FIG.13. There are eight 3-bit words stored. The contents of the eight tablelocations of the PCP remap LUT are the values in the right column. The3-bit address values that point to those table locations are indicatedin the left column.

FIG. 16 is a diagram that illustrates operation of a dual linked listsystem 200 involving link manager 122, linked list memory 129, freebuffer linked list memory 130, and packet descriptor and parse resultmemory 131. In one specific example, a queue element is a storageelement or pair of storage elements that stores: 1) a value, and 2) apointer that points to another queue element. This is but an example.There are other ways of implementing queues in hardware as explainedbelow (see, for example, FIG. 20 and the corresponding description), butthis simplified way is used here for instructional and illustrativepurposes in connection with the circuit of FIG. 16.

In the case of FIG. 16, each of memories 129 and 130 is a pipelinedmemory adapted to store queue elements. Each memory has a pipeline ofstages. In one example, memory 129 is such a memory system and memory130 is such a memory system. From the time a read request address issupplied to a memory system, a first clock cycle is required for addressinformation to propagate to a pipeline register on the way to thememory. A second clock cycle is required to pass out of the pipelineregister and to get set up on the address inputs of the internal memory.A third clock cycle is required for the data value to be output from theinternal memory. A fourth clock cycle is required for the output datavalue to pass through a pipeline register. A fifth clock cycle isrequired to perform error detection and correction on the data. A sixthclock cycle is required to set a valid bit (to indicate the memory hasoutput a valid data value). Accordingly, there are seven clock cycles ofdelay between the time a dequeue request is made from a requestingentity (for example, port dequeue engine 134) to the link manager 122until the time the data (as read from a queue element in the memory) isloaded into the popped head pointer queue element (the replenish thehead pointer queue element). Multiple reads of such a pipelined memorysystem can be taking place at the same time, with each stage operatingon a different read in a pipelined manner.

For each linked list maintained by link manager 122, the link manager122 maintains a first head pointer queue element H1, a second headpointer queue element H2, a first tail pointer queue element T1, asecond tail pointer queue element T2, a head pointer active bit, a tailpointer active bit, a buffer count B#, and a packet count P#. Each ofthese linked lists is actually a dual linked list involving: 1) a firstlinked list of queue elements (involving the first head pointer queueelement H1 and the first tail pointer queue element T1), and 2) arelated second linked list of queue elements (involving the second headpointer queue element H2 and the second tail pointer queue element T2).Into which of the two linked lists the next value (in this case, thenext buffer ID) will be pushed in a push operation is determined by thecontent of the tail pointer active bit. From push to push, the value ofthe tail pointer active bit toggles. From which of the two linked liststhe next value will be popped in a pop operation is determined by thecontent of the head pointer active bit. From pop to pop, the value ofthe head pointer active bit toggles. The link manager 122 can maintainup to sixty-four such dual linked lists of queue elements, as well as adual linked list of queue elements for the free buffer linked list.

An enqueue engine (for example, port enqueue engine 133 of FIG. 10) cancause a sequence of values (in this case, a sequence of buffer IDs) tobe enqueued into one of the dual linked lists such that the odd valuesof the sequence are enqueued by pushing the odd values into the firstlinked list of queue elements, and such that the even values of thesequence are enqueued by pushing the even values into the second linkedlist of queue elements. Values are pushed into the first and secondlinked lists in alternating fashion, with the content of the tailpointer active bit indicating which one of the two linked lists will bepushed next.

In this example, each of the two linked lists of a dual linked listoperates as follows. If the linked list is empty, then neither the tailpointer queue element nor the head pointer queue element stores anyvalue (any buffer ID). The next value that is pushed into the linkedlist is written into both the head and the tail pointer queue elements.If there is already one value stored in the linked list, and anothervalue is to be pushed, then the new value is pushed into the tailpointer queue element. The head pointer queue element is made to pointto the tail pointer queue element. At this point the head and tailpointer queue elements store different values, and two values are storedin the linked list. If there are already two values stored in the linkedlist in this way, and a third value is to be pushed, then the new valueis pushed into the tail pointer queue element, and the value that was inthe tail pointer queue element is stored in a third queue element inmemory. The head pointer queue element is made to point to the queueelement in memory, and the queue element in memory is made to point tothe tail queue element. If there are already three values being storedin the linked list, and fourth value is to be pushed into the linkedlink, then the value of the tail pointer is moved into memory to be thevalue of a second queue element in memory, and the new value is pushedinto the tail pointer queue value. The header pointer queue element ismade to point to the second queue element in memory, the second queueelement in memory is made to point to the first queue element in memory,and the first queue element in memory is made to point to the tailpointer queue element.

In this example, the opposite procedure is followed to pop a value off alinked list. The value of the head pointer queue element is output(popped), and the value of the queue element to which the head pointerqueue element had previously pointed is moved to the new head pointerqueue element. The head pointer queue element is changed so that itpoints to the next queue element in the linked list. If there are onlytwo values stored in the linked list, then those two values will bestored in the head and tail pointer queue elements, and the head pointerqueue element will point to the tail. If in that condition the linkedlist is to be popped, then the value of the head pointer is output(popped), and the value of the tail is copied to the head pointer queueelement such that both the head and tail pointer queue elements willstore the same value. In this condition, the linked list stores onevalue. If in this condition the linked list is to be popped, then thevalue of the head pointer queue element is output (popped) and both thehead and tail pointers are erased (both the head pointer queue elementand the tail pointer queue element are empty).

The dual linked list system 200 maintains two such linked lists for eachdual linked list. Popping a head pointer queue element takes arelatively small amount of time (for example, two clock cycles) ascompared to the larger amount of time required to read a value out ofthe memory to replenish the head pointer. If the dual linked list ispopped twice in rapid succession, then one of the head pointer queueelement is popped, and then the other head pointer queue element ispopped. During the time that the second head pointer queue element issupplying its output value, the memory can be updating a queue elementof the linked list associated with the first head pointer. Likewise, ifthe dual linked list is popped again, then the first head pointer willbe popped, and while the first head pointer is outputting its value thememory can be updating a queue element of the linked list associatedwith the second head pointer. Due to the pipelined nature of the memory,the rate at which a head pointer value can be popped off the dual linkedlist is smaller than the read access latency time of the memory. Thedual linked list system can be popped to output a new buffer ID everyother clock cycle, and the dual linked list system can pop a dual linkedlist repeatedly at a sustained rate to output two buffer IDs each sevenclock cycles. Each of the sixty-four linked lists for the sixty-fourvirtual channels is such a dual linked list. Also, the linked list forthe free buffer linked list is such a dual linked list. As describedabove, the values stored in these linked lists are buffer IDs.

FIG. 16 illustrates a condition in which a dual linked list of eightqueue elements stores eight values (eight buffer IDs) for the firstvirtual channel. The first value in the sequence of values to be pushedinto the dual linked list is stored in QE1. The second value in thesequence is stored in QE2, and so forth. The first head pointer queueelement H1 stores the first value, and points to the queue element QE3in memory that stores the third value. This queue element QE3 points tothe queue element Q5 in memory that stores the fifth value. This queueelement QE5 points to the T1 tail queue element that stores the seventhvalue. The tail pointer queue element does not pointer anywhere becauseit is the tail. That is the first linked list of the dual linked list.The second linked list of the dual linked list involves the second valuethat is stored in the H2 head pointer queue element QE2. The H2 queueelement (QE2) points to the queue element Q4 in memory that stores thefourth value. This queue element QE4 points to the queue element QE6 inmemory that stores the sixth value. This queue element QE6 points to theT2 tail pointer queue element (QE8) that stores the eighth value. The T2tail pointer queue element does not point anywhere because it is a tailpointer queue element. According the first value of the sequence isstored in QE1, the second value of the sequence is stored in QE2, thethird value of the sequence is stored in QE3, and so forth. If thelinked list is to be popped, then the values of the sequence will beread out of the head pointer queue elements in the same order. The firstvalue will be popped first, the second value will be popped second, andso forth. In addition to this dual linked list maintained for the firstvirtual channel, a second dual linked list is maintained for the freebuffer linked list. In the case of the free buffer linked list, the linkmanager 122 maintains a free buffer head FIFO 181. If, for example, thefree buffer head FIFO 181 can store four values (four buffer IDs), thenthe link manager 122 keeps the head pointer queue elements popped suchthat the next four head pointer values (that could be popped) arepreloaded from the linked list and are sitting in the free buffer headFIFO. The read access time for reading the free buffer head FIFO issmaller than the read access time of the free buffer linked list memory130, so four successive buffer IDs can be supplied by the overall freebuffer dual linked list in rapid succession with the buffer IDs that areoutput being drawn from the free buffer head FIFO.

FIG. 17 is a diagram that illustrates the two linked lists of the duallinked list for the first virtual channel, as explained above. The firstlinked list involves the queue elements QE1, QE3, QE5 and QE7 storingthe odd values of the sequence of values. The second linked listinvolves the queue elements QE2, QE4, QE6 and QE8 storing the even valueof the sequence of values. Values are pushed onto the tail pointer queueelements, and are popped off the head pointer queue elements. The“values” are buffer IDs.

FIG. 18 is a diagram that illustrates how a value (a buffer ID) ispopped off a dual linked list. A port dequeue engine 134 issues adequeue request to the link manager 122 along with a virtual channelnumber. The virtual channel number indicates which one of multiple duallinked lists is to be popped. The active head pointer queue element (asindicated by the head pointer active bit) is popped and the obtainedbuffer ID is returned to the port dequeue engine 134. The linked list(that involves the head pointer queue element that was just popped)must, however, be updated. The queue element in the memory to which thehead pointer queue element had pointed must be read from memory, andthat value must then be loaded into the head pointer queue element thatwas just popped. To do this, a read request 206 is issued to pipelinedmemory system 129. A first clock cycle is required by the link manager122 to respond to the dequeue request and to get a read address outputto a pipeline register 201 on the way to the pipelined memory system. Asecond clock cycle is required to pass out of the pipeline register 201and to get set up on the address inputs of the internal memory 202. Athird clock cycle is required for the data value to be output from theinternal memory 202. A fourth clock cycle is required to pass through apipeline register 203. A fifth clock cycle is required to for an ECCcircuit 204 to perform error detection and correction on the data. Asixth clock cycle is required to pass through an output register 205 andto set a valid bit (to indicate a valid data value is available on theoutputs of the pipelined memory system). From the time the dequeuerequest is supplied from the port dequeue engine 134 to the link manager122 until the time the read data (the buffer ID) from the pipelinedmemory 129 is loaded into the head pointer queue element (the headpointer queue element that was just popped) is seven clock cycles. Thepointer of the head pointer queue element in the link manager 122 ischanged so that it contains the pointer of the queue element just readout of memory. The result is that the head pointer queue elementcorrectly points to the next queue element in the appropriate linkedlist. When this is done, the head pointer queue element is said to be“replenished”.

FIG. 19 is a simplified waveform diagram that illustrates two headpointer queue element pops done one after the other in rapid successionfor the same virtual channel, in the case of dequeueing two consecutiveframes that are of the minimum frame size (sixty-four bytes). From thetime of the first dequeue request a time T1 until the time when the readbuffer ID has replenished the popped head pointer queue element at timeT8 is seven clock cycles. Nonetheless, due to pipelining in the memory,two head pointer values can be popped from the link manager 122 everyseven clock cycles, repeatedly, and indefinitely. As shown in FIG. 19,multiple read assess of the pipelined memory are occurring at the sametime, with one read access passing through the stages of the pipelinememory a few clock cycles behind the other.

The term “queue element” as it is used in this patent document is abroad term. There are multiple ways of implementing a queue element. Aqueue element need not necessarily involve a storage location in amemory or other sequential logic element, where both the value of thequeue element is stored and also the pointer of the queue element isstored. In some examples of a queue element, the association of thepointer of the queue element and the value of the queue element ismaintained by the way values are stored in a memory and by the way thatmemory is addressed. For example, the value of a first queue element mayitself be used as the pointer of the queue element, where the pointer ofthe queue element is used as an address into the memory to address thenext queue element that is being pointed to. The address locationpointed to is made to store the value of the next queue element.

FIGS. 20A-20E together form a larger diagram, FIG. 20. FIG. 20illustrates an example of pushing and popping one of the two linkedlists of the dual linked list for the first virtual channel, for thenovel dual linked list of buffers described above. Initially, in thisexample, the linked list empty. The head pointer queue element H1 isempty and the tail pointer queue element T1 is empty. Next, a firstvalue is pushed. In this example, the values pushed are bufferidentification values (buffer IDs). The first value B1 is pushed, bypushing the value B1 into the tail pointer queue element T1. Ordinarily,the value being pushed (in this case B2) would then also be written intothe memory at the location pointed to by the old tail pointer value, buthere the old tail pointer value is “empty” so there is no memory write.In addition, the value being pushed (B2) is stored into the head pointerqueue element H1. At this point, both the head and tail pointer queueelements store the same value B1 and the linked list (the “queue”)stores one value, B1. In this pipelined memory a write can be performedin one clock cycles, whereas a read requires seven clock cycles (fromthe time a read request is made until the data read is returned from thepipelined memory).

Next, the second value is pushed. The value B2 is loaded into the tailpointer queue element T1, and in addition the new value being pushed(B2) is written into the memory at the location pointed to by the oldtail pointer value (B1). At this point, the head pointer queue elementH1 stores the value B1, the tail pointer queue element T1 stores thevalue B2, and the value B2 is stored in the memory at the locationpointed to by the value B1. The linked list stored two values.

Next, a third value (B3) is pushed. The value B3 is loaded into the tailpointer queue element T1, and in addition the new value being pushed(B3) is written into the memory at the location pointed to by the oldtail pointer value (B2). At this point, the head pointer queue elementH1 stores the value B1, the tail pointer queue element T1 stores thevalue B3, the memory stores the value B2 at the location pointed by B1,and the memory stores the value B3 at the location pointed to by B2. Thelinked list stores three values.

Next, a fourth value (B4) is pushed. The value B4 is loaded into thetail pointer queue element T1, and in addition the new value beingpushed (B4) is written into the memory at the location pointed to by theold tail pointer value (B3). At this point, the head pointer queueelement H1 stores the value B1, the tail pointer queue element T1 storesthe value B4, the memory stores the value B2 at the location pointed byB1, and the memory stores the value B3 at the location pointed to by B2,and the memory stores the value B4 at the location pointed to by B3. Thelinked list stores four values.

Next, in this example, the linked list is popped. The value B1 stored inthe head pointer queue element H1 is output. To replenish the headpointer queue element H1 so that it stores the next value to be popped,the memory is read at the location pointed to by the old head pointervalue (B1). The value stored in memory at the location addressed by thevalue B1 is the value B2. This value B2 is therefore read from memory,and is loaded into the head pointer queue element H1. At this point, thetail pointer queue element T1 stores the value B4, the head pointerqueue element H1 stores the value B2, the memory stores the value B3 atthe location pointed by the value B2, and the memory stores the value B4at the location pointed to by the value B3. The linked list stores threevalues.

Next, the linked list is popped for the second time. The value B2 storedin the head pointer queue element H1 is output. To replenish the headpointer queue element H1 so that it stores the next value to be popped,the memory is read at the location pointed to by the old head pointervalue (B2). The value stored in memory at the location addressed by thevalue B2 is the value B3. This value B3 is therefore read from memory,and is loaded into the head pointer queue element H1. At this point, thetail pointer queue element T1 stores the value B4, the head pointerqueue element H1 stores the value B3, and the memory stores the value B4at the location pointed to by the value B3. The linked list stores twovalues.

Next, the linked list is popped for the third time. The value B3 storedin the head pointer queue element H1 is output. To replenish the headpointer queue element H1 so that it stores the next value to be popped,the memory is read at the location pointed to by the old head pointervalue (B3). The value stored in memory at the location addressed by thevalue B3 is the value B4. This value B4 is therefore read from memory,and is loaded into the head pointer queue element H1. At this point, thetail pointer queue element T1 stores the value B4, and the head pointerqueue element H1 also stores the value B4. The linked list stores onevalue.

Next, the linked list is popped for the fourth time. The value B4 storedin the head pointer queue element H1 is output. Because the valuesstored in the head pointer queue element H1 and the tail pointer queueelement T1 were the same prior to the fourth pop, the head and tailpointer values are now made to be empty. The linked list is empty.Accordingly, in the example of FIG. 20, each queue element is a singlestorage location.

FIG. 21 is a diagram of the form of an IEEE 802.3x Priority Flow Control(PFC) pause frame 210, such as might be output by the NFP integratedcircuit 9 when one or more virtual channels are overloaded. In the caseof PCP flow merging happening in the port enqueue engine responsible forhandling ingress through the physical MAC port (the port through whichthe overloading frames were received), the PFC pause frame 210 is outputin the opposite direction on that same physical MAC port. The PFC pauseframe is a request that the sender of the ethernet frames pause insending frames on the physical MAC port if those frames have specifiedPCP code values. In the PFC pause frame, multiple ones of the enablebits in the priority class enable vector are set if PCP flow merging isemployed, and the merged flow is to be paused. For example, in the caseof the PCP flow merging of FIG. 14, and in the case of overloading ofthe virtual channel into which the three PCP flows (for PCP values 5, 6and 7) are merged, the enable bits EN[5], EN[6] and EN[7] in the PFCpause frame are set. The setting of these enable bits means that a pauserequest is being made: 1) to pause flows having a PCP code value of 5for a pause time specified by a corresponding one of the pause timefields denoted “PAUSE TIME (P CLASS 5)”, 2) to pause flows having a PCPcode value of 6 for a pause time specified by a corresponding one of thepause time fields denoted “PAUSE TIME (P CLASS 6)”, and 3) to pauseflows having a PCP code value of 7 for a pause time specified by acorresponding one of the pause time fields denoted “PAUSE TIME (P CLASS7)”. All three PCP flows that are being merged into the one overloadedvirtual channel are all requested to be paused at the same time due tothe same one PFC pause frame 210. To generate the pause frame, the portenqueue engine outputs the enable bits on conductors 175 of FIG. 10A. Inresponse, the appropriate physical MAC port (configured portion of theEthernet MAC portion 123) handles generating the PFC pause frame andoutputting the PFC pause frame on the physical MAC port.

FIG. 22 is a flowchart of a method that involves inverse PCP flowremapping in accordance with one novel aspect. Multiple frames arereceived (step 301) onto a physical MAC port of the NFP integratedcircuit. Configuration information is supplied (step 302) to an “InversePCP Remap LUT” (IPRLUT) circuit within the NFP integrated circuit. Framedata is written (step 303) into a linked list of buffers, where thelinked list of buffers stores frames for a single (one and only one)virtual channel. A buffer count is maintained (step 304) for the linkedlist of buffers. As the frames are received, the frames data is storedin the linked list of buffers and the buffer count goes up and downdepending on how the enqueue engine and the dequeue engine handling thevirtual channel are operating. The NFP integrated circuit also stores(step 305), for the virtual channel, a predetermined overflow thresholdvalue. This predetermined overflow value may be part of theconfiguration information and/or may be supplied by another monitoringprocessor. In this example, at a point in the receiving of frames, thelink manager determines (step 306) that the buffer count has exceededthe predetermined overflow threshold value. As a result of thedetermining that the buffer count has exceeded the predeterminedthreshold value, the virtual channel number is supplied (step 307) tothe IPRLUT. In response, the IPRLUT outputs a multi-bit value. Theinverse lookup function is determined by the configuration informationloaded into the IPRLUT in step 302. The multi-bit value includes aplurality of bits, where each bit corresponds to a PCP code prioritylevel. In the enqueuing operation, PCP merging was performed.Accordingly, multiple ones of the bits in the multi-bit value (output bythe IPRLUT) are set, thereby indicating that not just one but rathermultiple PCP flows should be paused. Even though only one virtualchannel may be overloaded, multiple PCP flows are to be paused. Themulti-bit value as output by the IPRLUT is then used (step 308) togenerate a PFC pause frame, where multiple ones of the enable bits inthe priority class enable vector of the PFC pause frame are set. FIG. 21is a diagram of the format of the PFC pause frame. In the example ofFIGS. 9 and 10, the Ethernet MAC portion 123 performs this operation ofgenerating the PFC pause frame from the multi-bit value. The resultingPFC pause frame is then output (step 309) from the physical MAC port ofthe NFP integrated circuit.

In one example, to realize an integrated circuit embodiment of theingress MAC island circuitry of FIG. 5, the function of the each circuitblock is described in a hardware description language (for example,Verilog or VHDL or CDL). A hardware description language description ofEthernet MAC portion 123 is obtained from MorethanIP GmbH, MuenchnerStrasse 199, D-85757 Karlsfeld, Germany, and is incorporated into theoverall hardware description. A commercially available hardwaresynthesis program (for example, Synopsis Design Compiler) is thenemployed to generate digital logic circuitry from the hardwaredescription language description, where the synthesized digital logiccircuitry performs the function described by the hardware descriptionlanguage.

Although certain specific embodiments are described above forinstructional purposes, the teachings of this patent document havegeneral applicability and are not limited to the specific embodimentsdescribed above. Accordingly, various modifications, adaptations, andcombinations of various features of the described embodiments can bepracticed without departing from the scope of the invention as set forthin the claims.

What is claimed is:
 1. A method comprising: (a) receiving configurationinformation onto a Media Access Control (MAC) layer interface circuit ofa Network Flow Processor (NFP) integrated circuit, wherein theconfiguration information includes port definition configurationinformation and Priority Code Point (PCP) remap information, wherein thePCP remap information includes a plurality of portions; (b) using theport definition configuration information to configure the MAC layerinterface circuit to include a first number of physical MAC ports; (c)receiving a plurality of PCP flows of ethernet frames via the physicalMAC ports onto the NFP integrated circuit, wherein all the frames of aPCP flow are received via the same physical MAC port and wherein all ofthe frames of the PCP flow have the same PCP value; (d) storing eachrespective portion of the PCP remap information in association with acorresponding respective one of the physical MAC ports; and (e) for eachframe received via a particular physical MAC port using the PCP value ofthe frame and the portion of the PCP remap information associated withthe physical MAC port to assign the frame to one of a second number ofvirtual channels, wherein the assigning of (e) involves assigning morethan one PCP flow to the same virtual channel, wherein each PCP flowreceived in (c) via a particular physical MAC port has a relativepriority with respect to each other PCP flow received in (c) via thephysical MAC port, wherein the relative priorities of PCP flows withinthe particular physical MAC port are maintained with respect to oneanother after the assigning of (e) even though some PCP flows may beassigned to the same virtual channel, and wherein the first numbermultiplied by eight is greater than the second number.
 2. The method ofclaim 1, further comprising: (f) maintaining a plurality of linked listsof buffers on the NFP integrated circuit, wherein there is a differentlinked list of buffers for each corresponding one of the plurality ofvirtual channels, and wherein the assigning of a PCP flow to a virtualchannel in (e) involves storing frames of the PCP flow into the linkedlist of buffers for the virtual channel.
 3. The method of claim 2,wherein the frames of the plurality of PCP flows received in (c) afterbeing assigned to virtual channels in (e) are communicated out of theMedia Access Control (MAC) layer interface circuit across a single bus.4. The method of claim 1, wherein the PCP flows of the more than one PCPflow that are assigned to the same virtual channel in (e) are a firstset of PCP flows that are received via the same physical MAC port andthat have PCP values of a contiguous block of PCP values, wherein no PCPflow that is received onto said same physical MAC port and that has aPCP value in the contiguous block is also assigned in (e) to a differentvirtual channel.
 5. The method of claim 1, wherein (e) further involvesassigning a second set of PCP flows to the same second virtual channel,wherein the second set of PCP flows are received via the same physicalMAC port via which the plurality of PCP flows of (c) are received. 6.The method of claim 1, wherein the NFP integrated circuit isconfigurable such that a particular physical MAC port of the NFPintegrated circuit is configurable to be either a 10 Gbps (gigabits persecond) port, a 40 Gbps port, or a 100 Gbps port, and wherein how theparticular physical MAC port is configured is determined by the portdefinition configuration information.
 7. The method of claim 1, whereinthe NFP integrated circuit comprises a plurality of Look Up Table (LUT)circuits, wherein each respective one of the LUT circuits stores aportion of the PCP remap information for a corresponding respective oneof the physical MAC ports, and wherein each LUT circuit converts anincoming three-bit PCP value into an outgoing three-bit number.
 8. Themethod of claim 1, wherein the configuration information received in (a)is received onto the MAC layer interface circuit from another part ofthe NFP integrated circuit.
 9. A method comprising: (a) maintaining aplurality of linked lists of buffers in a Media Access Control (MAC)layer interface circuit of a Network Flow Processor (NFP) integratedcircuit; (b) receiving configuration information onto the MAC layerinterface circuit of the NFP integrated circuit, wherein theconfiguration information includes port definition configurationinformation and Priority Code Point (PCP) remap information, wherein thePCP remap information includes a plurality of portions; (c) using theport definition configuration information to configure the MAC layerinterface circuit to include a first number of physical MAC ports; (d)receiving a plurality of PCP flows of ethernet frames via the physicalMAC ports onto the NFP integrated circuit, wherein all the frames of aPCP flow are received via the same physical MAC port and wherein all ofthe frames of the PCP flow have the same PCP value; (e) storing eachrespective portion of the PCP remap information in association with acorresponding respective one of the physical MAC ports; and (f) for eachframe received via a particular physical MAC port using the PCP value ofthe frame and the portion of the PCP remap information stored inassociation with the physical MAC port to assign the frame to one of thelinked lists, wherein the assigning of (f) involves assigning more thanone PCP flow to the same linked list, and wherein there are more PCPflows received in (d) than there are linked lists maintained in (a). 10.The method of claim 9, wherein each PCP flow received in (d) via aparticular physical MAC port has a relative priority with respect toeach other PCP flow received in (d) via the same particular physical MACport, wherein the relative priorities of PCP flows within the particularphysical MAC port are maintained with respect to one another after theassigning of (f) even though some of the PCP flows may be assigned tothe same linked list of buffers.
 11. The method of claim 10, furthercomprising: (g) operating a plurality of virtual channels in the NFPintegrated circuit, wherein each virtual channel is a packet processingchannel, and wherein there is a one-to-one correspondence between thelinked lists of plurality of linked lists and the virtual channels ofthe plurality of virtual channels.
 12. The method of claim 11, furthercomprising: (g) after the assigning of (f) transferring the frames ofthe plurality of PCP flows from the linked lists of buffers and out ofthe Media Access Control (MAC) layer interface circuit across a singlebus.
 13. The method of claim 12, wherein the NFP integrated circuitcomprises a plurality of Look Up Table (LUT) circuits, wherein eachrespective one of the LUT circuits stores a portion of the PCP remapinformation for a corresponding respective one of the physical MACports, and wherein each LUT circuit converts an incoming three-bit PCPvalue into an outgoing three-bit number.
 14. A Network Flow Processor(NFP) integrated circuit, comprising: a Media Access Control (MAC) layerinterface circuit that is configurable to include a plurality ofphysical MAC ports, wherein each physical MAC port can receive a PCPflow of ethernet frames onto the NFP integrated circuit, wherein all theframes of a PCP flow are received via the same physical MAC port andwherein all of the frames of the PCP flow have the same Priority CodePoint (PCP) value; a memory that stores a plurality of linked lists ofbuffers; a plurality of port enqueue engines, wherein each respectiveone of the port enqueue engines is configured to receive ethernet framesfrom a respective corresponding one of the physical MAC ports and tocause the frames to be stored into one or more of the linked list ofbuffers, wherein each port enqueue engine is configurable to merge PCPflows of frames such that frames of more than one PCP flow are stored inthe same linked list of buffers in the memory, wherein each PCP flowreceived onto the NFP integrated circuit via a particular physical MACport has a relative PCP priority with respect to each other PCP flowreceived via the same particular physical MAC port, wherein the relativePCP priorities of PCP flows within the particular physical MAC port aremaintained with respect to one another after a port enqueue port enginehas caused the frames to be stored in the linked lists of buffers eventhough some of the PCP flows have been merged so that their frames arestored in the same linked list of buffers, wherein the number ofphysical MAC ports multiplied by eight is greater than the number oflinked lists of buffers; an output bus; and a plurality of port dequeueengines, wherein each respective one of the port dequeue enginesreceives frames from the memory and outputs the frames so that theframes are then communicated out of the MAC interface circuit via theoutput bus.
 15. The NFP integrated circuit of claim 14, wherein the NFPintegrated circuit stores multiple portions of PCP remap information,wherein there is one portion of PCP remap information for eachrespective one of the port enqueue engines, wherein each port enqueueengine uses its portion of the PCP remap information along with a PCPvalue of a frame to assign the frame to one of the linked lists, whereinthe NFP integrated circuit further stores port definition configurationinformation, and wherein the port definition configuration informationdetermines a configurable number of physical MAC ports into which theMAC layer interface circuit is configured.
 16. The NFP integratedcircuit of claim 15, wherein each port enqueue engine includes a PCPremap Look Up Table (LUT) circuit, and wherein each respective portionof the PCP remap information is a number of bits stored in acorresponding respective one of the PCP remap LUTs.
 17. The NFPintegrated circuit of claim 14, wherein each physical MAC port outputsframes along with a port number, and wherein each of the port enqueueengines only processes frames that are output by a physical MAC portalong with one particular port number.
 18. The NFP integrated circuit ofclaim 14, wherein each physical MAC port outputs an amount of frame dataalong with a port number, and wherein each of the port enqueue enginesonly processes an amount of frame data if the amount of frame data has aparticular port number associated with the port enqueue engine, whereineach port enqueue engine comprises: a PCP detecting circuit thatidentifies a three-bit PCP value in an amount of frame data; a PCP remapcircuit that receives the three-bit PCP value and performs PCP remappingthereby outputting a three-bit remapped PCP value, wherein the PCP remapcircuit is configurable so that it can be configured to output the samethree-bit remapped PCP value for multiple different three-bit PCP valuesreceived from the PCP detecting circuit; an adder that adds the remappedPCP value to a base value thereby outputting a channel number; and meansfor using the channel number to obtain a buffer identification value,wherein the buffer identification value identifies a buffer of a linkedlist of buffers, and wherein the means is also for causing the amount offrame data to be written into the buffer.
 19. The NFP integrated circuitof claim 18, wherein the NFP integrated circuit further comprises a linkmanager, and wherein in the means comprises: a circuit that suppliesdequeue requests to the link manager.
 20. The NFP integrated circuit ofclaim 18, wherein the PCP remap circuit is a LUT (Look Up Table)circuit.